Configuring a system to collect and aggregate datasets

ABSTRACT

Methods for configuring a system to collect and aggregate datasets are disclosed. One embodiment includes, identifying a data source in the system from where dataset is to be collected, configuring a machine in the system that generates the dataset to be collected, to send the dataset to the data source, identifying an arrival location where the dataset that is collected is to be aggregated or written, and/or configuring an agent node by specifying a source for the agent node as the data source in the system and specifying a sink for the agent node as the arrival location.

CLAIM OF PRIORITY

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/319,816 entitled “CONFIGURING A SYSTEM TO COLLECT AND AGGREGATEDATASETS”, which was filed on Mar. 31, 2010, Attorney Docket No.68784-8002.US00, the contents of which are expressly incorporated byreference herein.

BACKGROUND

Increasingly large amounts of data are generated with expanded user baseof online applications such as social networks. Online activity of usersinteracting with one another, or with applications generates valuableinformation about users. Such data and analytics therein extracted canpotentially provide service providers with useful information, forexample, regarding traffic, content popularity, trends, etc.Additionally, advertisers can potentially glean information regardinguser information and interests.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of client devices that generatedatasets (log data) to be collected and aggregated for storage in thestorage device via interacting nodes in various tiers, in someinstances, through a network.

FIG. 2A depicts a diagram showing one example configuration of nodesthat are controlled/configured by a master and dataflow therein from adata source to a data sink (e.g., storage).

FIG. 2B depicts diagrams showing examples of configuration of nodescontrolled/configured by a master and the dataflow therein from a datasource to a data sink.

FIG. 3A depicts a diagram showing one example configuration of a node(nodes) controlled by a master residing on the same machine and dataflowtherein from a data source to a data sink.

FIG. 3B depicts a diagram showing one example configuration of a nodecontrolled by a master residing on a different machine and dataflowtherein from a data source to a data sink.

FIG. 4 depicts a diagram showing an example configuration of nodes inmultiple tiers on the same machine controlled by multiple mastersresiding on different machines and dataflow therein from data sources toa data sink.

FIG. 5 depicts a diagram showing an example configuration of nodes inmultiple tiers on different machines controlled by multiple mastersresiding on different machines and dataflow therein from data sources toa data sink.

FIG. 6 depicts a block diagram illustrating example components of a nodethrough which dataflow occur.

FIG. 7 depicts a block diagram illustrating example components of adecorator module in a node.

FIG. 8 depicts a block diagram illustrating example components of amaster that controls the nodes in the system.

FIG. 9 depicts an example of an extensible data model used to representevents or data flow in a distributed system from a source to a sink.

FIG. 10A depicts a screenshot of an example user interface showing nodestatus and configurations with node mappings.

FIG. 10B depicts a screenshot of an example user interface forconfiguring a node.

FIG. 10C depicts a screenshot of an example user interface forconfiguring multiple nodes.

FIG. 10D depicts a screenshot of an example user interface for issuingraw commands.

FIG. 10E depicts a screenshot of an example user interface showingstatistics for a node.

FIG. 10F depicts a screenshot of an example user interface showingstatistics for a node.

FIG. 11 depicts a flowchart of an example process of facilitatingcollecting and aggregating datasets that are machine or user-generatedfor analysis.

FIG. 12 depicts a flowchart of an example process for configuring asystem to collect and aggregate datasets.

FIG. 13 depicts a flowchart of an example for scaling the system byimplementing multiple masters on multiple machines.

FIG. 14 depicts a flowchart of another example process for configuring asystem to collect and aggregate datasets and to perform analytics on thedatasets.

FIG. 15 depicts a flowchart of an example process for dynamicallyprocessing an event including a dataset that is streamed from a sourceto a sink via nodes.

FIG. 16 depicts a flowchart of an example process for collecting andaggregating datasets for storage in a file system with fault tolerance.

FIG. 17 depicts a flowchart of an example process for deleting the WALin response to acknowledgement of receipt by the master.

FIG. 18 depicts a flowchart of an example process for collecting andaggregating datasets with fault tolerance using a store on failuremechanism.

FIG. 19 shows a diagrammatic representation of a machine in the exampleform of a computer system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

DETAILED DESCRIPTION

The following description and drawings are illustrative and are not tobe construed as limiting. Numerous specific details are described toprovide a thorough understanding of the disclosure. However, in certaininstances, well-known or conventional details are not described in orderto avoid obscuring the description. References to one or an embodimentin the present disclosure can be, but not necessarily are, references tothe same embodiment; and, such references mean at least one of theembodiments.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described which may be requirementsfor some embodiments but not other embodiments.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the disclosure, and in thespecific context where each term is used. Certain terms that are used todescribe the disclosure are discussed below, or elsewhere in thespecification, to provide additional guidance to the practitionerregarding the description of the disclosure. For convenience, certainterms may be highlighted, for example using italics and/or quotationmarks. The use of highlighting has no influence on the scope and meaningof a term; the scope and meaning of a term is the same, in the samecontext, whether or not it is highlighted. It will be appreciated thatsame thing can be said in more than one way.

Consequently, alternative language and synonyms may be used for any oneor more of the terms discussed herein, nor is any special significanceto be placed upon whether or not a term is elaborated or discussedherein. Synonyms for certain terms are provided. A recital of one ormore synonyms does not exclude the use of other synonyms. The use ofexamples anywhere in this specification including examples of any termsdiscussed herein is illustrative only, and is not intended to furtherlimit the scope and meaning of the disclosure or of any exemplifiedterm. Likewise, the disclosure is not limited to various embodimentsgiven in this specification.

Without intent to further limit the scope of the disclosure, examples ofinstruments, apparatus, methods and their related results according tothe embodiments of the present disclosure are given below. Note thattitles or subtitles may be used in the examples for convenience of areader, which in no way should limit the scope of the disclosure. Unlessotherwise defined, all technical and scientific terms used herein havethe same meaning as commonly understood by one of ordinary skill in theart to which this disclosure pertains. In the case of conflict, thepresent document, including definitions will control.

Embodiments of the present disclosure include systems and methods forconfiguring a system to collect aggregate datasets.

FIG. 1 illustrates a block diagram of client devices 102A-N thatgenerate datasets (log data) to be collected and aggregated for storagein the storage device 108 via interacting nodes in various tiers, insome instances, through a network 106.

The client devices 102A-N can be any system and/or device, and/or anycombination of devices/systems that is able to establish a connectionwith another device, a server and/or other systems. The client devices102A-N typically include display or other output functionalities topresent data exchanged between the devices to a user. For example, theclient devices and content providers can be, but are not limited to, aserver desktop, a desktop computer, a thin-client device, an internekiosk, a computer cluster, a mobile computing device such as a notebook,a laptop computer, a handheld computer, a mobile phone, a smart phone, aPDA, a Blackberry device, a Treo, and/or an iPhone, etc. In oneembodiment, the client devices 102A-N are coupled to a network 106. Insome embodiments, the client devices may be directly connected to oneanother.

In one embodiment, users interact with user devices 102A-N (e.g.,machines or devices). As a results of the user interaction, the devices102A-N can generate datasets such as log files to be collected andaggregated. The file can include logs, information, and other metadataabout clicks, feeds, status updates, data from applications, andassociated properties and attributes.

User devices 102A-N can have nodes executing or running thereon thatcollect the datasets that are user-generated or machine-generated, forexample, based on user-interaction with applications or websites runningon the devices. Such nodes can interact and/or communicate with one ormore other nodes (e.g., either running on the same device/machine oranother device/machine (e.g., machine/device 104) to facilitatecollection and aggregation of datasets thus generated. In oneembodiment, the datasets are eventually written to a file and stored,for example, in storage (e.g., repository 130) on a physical disk.

Additionally, functionalities and properties can be assigned to thenodes such that various analytics can be performed on the collecteddataset and additional information can be extracted or embedded. Thedataflow among nodes can be configured at a master. In one embodiment,the nodes executed on the machines 102 or 104 can contact the master(s)to obtain configuration information, which have been set by default orconfigured by a user The master can be executed on the same devices102A-N, 104, or at the host device 100. One or multiple masters can beinvolved in the mapping of data flow among the nodes and variousmachines.

Some examples of architectural configurations among nodes, machines, andmasters are illustrated and described with further reference to theexample of FIG. 2-5. The functionalities of the nodes and configurationare described with further reference to the examples of FIG. 6-8.

The network 106, over which the client devices 102A-N, 104, host, andthe nodes and masters therein communicate may be a telephonic network,an open network, such as the Internet, or a private network, such as anintranet and/or the extranet. For example, the Internet can provide filetransfer, remote log in, email, news, RSS, and other services throughany known or convenient protocol, such as, but is not limited to theTCP/IP protocol, Open System Interconnections (OSI), FTP, UPnP, iSCSI,NSF, ISDN, PDH, RS-232, SDH, SONET, etc.

The network 106 can be any collection of distinct networks operatingwholly or partially in conjunction to provide connectivity to the clientdevices, host server, and may appear as one or more networks to theserviced systems and devices. In one embodiment, communications to andfrom the client devices 102A-N can be achieved by, an open network, suchas the Internet, or a private network, such as an intranet and/or theextranet. In one embodiment, communications can be achieved by a securecommunications protocol, such as secure sockets layer (SSL), ortransport layer security (TLS).

The term “Internet” as used herein refers to a network of networks thatuses certain protocols, such as the TCP/IP protocol, and possibly otherprotocols such as the hypertext transfer protocol (HTTP) for hypertextmarkup language (HTML) documents that make up the World Wide Web (theweb). Content is often provided by content servers, which are referredto as being “on” the Internet. A web server, which is one type ofcontent server, is typically at least one computer system which operatesas a server computer system and is configured to operate with theprotocols of the World Wide Web and is coupled to the Internet. Thephysical connections of the Internet and the protocols and communicationprocedures of the Internet and the web are well known to those of skillin the relevant art. For illustrative purposes, it is assumed thenetwork 106 broadly includes anything from a minimalist coupling of thecomponents illustrated in the example of FIG. 1, to every component ofthe Internet and networks coupled to the Internet.

In addition, communications can be achieved via one or more wirelessnetworks, such as, but is not limited to, one or more of a Local AreaNetwork (LAN), Wireless Local Area Network (WLAN), a Personal areanetwork (PAN), a Campus area network (CAN), a Metropolitan area network(MAN), a Wide area network (WAN), a Wireless wide area network (WWAN),Global System for Mobile Communications (GSM), Personal CommunicationsService (PCS), Digital Advanced Mobile Phone Service (D-Amps),Bluetooth, Wi-Fi, Fixed Wireless Data, 2G, 2.5G, 3G networks, enhanceddata rates for GSM evolution (EDGE), General packet radio service(GPRS), enhanced GPRS, messaging protocols such as, TCP/IP, SMS, MMS,extensible messaging and presence protocol (XMPP), real time messagingprotocol (RTMP), instant messaging and presence protocol (IMPP), instantmessaging, USSD, IRC, or any other wireless data networks or messagingprotocols.

The client devices 102A-N can be coupled to the network (e.g., Internet)via a dial up connection, a digital subscriber loop (DSL, ADSL), cablemodem, and/or other types of connection. Thus, the client devices 102A-Ncan communicate with remote servers (e.g., web server, host server, mailserver, and instant messaging server) that provide access to userinterfaces of the World Wide Web via a web browser, for example.

The repository 130 can store software, descriptive data, images, systeminformation, drivers, collected datasets, aggregated datasets, logfiles, analytics of collected datasets, enriched datasets, etc. Therepository may be managed by a database management system (DBMS), forexample but not limited to, Oracle, DB2, Microsoft Access, Microsoft SQLServer, MySQL, FileMaker, etc.

The repositories can be implemented via object-oriented technologyand/or via text files, and can be managed by a distributed databasemanagement system, an object-oriented database management system(OODBMS) (e.g., ConceptBase, FastDB Main Memory Database ManagementSystem, JDOlnstruments, ObjectDB, etc.), an object-relational databasemanagement system (ORDBMS) (e.g., Informix, OpenLink Virtuoso, VMDS,etc.), a file system, and/or any other convenient or known databasemanagement package. In one embodiment, the repository is managed by adistributed file system or network file system that allows access tofiles from multiple hosts/machines over a network. The distributed filesystem can include by way of example, the Hadoop Distributed File system(HDFS). Other file systems can be used as well, for example, throughintegration of Hadoop's interface which provides an abstraction layerfor the file system. For example, a local file system where a noderesides can be used. The HDFS native distributed file system can also beused. In addition, S3 (a remote file system hosted by Amazon webservices), FTP, and KFS (Kosmos file system—another distributed filesystem) can also be used. Clients can also write to different filesystems (NFS), or other file systems.

In general, the user devices 102 and 104 are able to write files (e.g.,files including by way of example, collected and aggregateddatasets/logs/log files) to the repository 130, either through thenetwork 106 or without utilizing the network 106. The host server 100can be implemented on a known or convenient computer system, such as isillustrated in FIG. 19.

FIG. 2A depicts a diagram showing one example configuration 200 of nodes202 and 204 that are controlled/configured by a master 206 and dataflowtherein from a data source 201 to a data sink 207 (e.g., repository208).

In order to collect dataset on a machine for aggregation and to performany analytics, the node 202 can be mapped to the data source 201 on themachine that generates that dataset. The dataset can include data,metadata, complex data, including logs of clicks, social networkingsites, feeds, status updates, logs from local or remote applications,etc. The data source can include, by way of example, but not limitation,a syslog port, an incoming network connection, and an IRC channel,output from execution of a program, a text file. In addition, a datasource can include a Hadpp sequence file formatted file.

The node 202 can map the data source 201 to a receiving location such asa data sink 203. In one embodiment, a master 206 controls the mapping ofthe data source 201 to the receiving location 203 via the node 202. Inaddition, data flow of the dataset from the data source 201 to thereceiving location 203 is configurable and reconfigurable at the master206 by a user (e.g., system administrator, developer, etc.). The node202 can contact the master 206 to obtain its mapping/configurationinformation.

Having nodes contact the master 206 to obtain configuration informationallows node configuration to dynamically change without having to loginand to restart the daemon (e.g., a server process on a UNIX-basedmachine). The node's configuration can be changed to a new one withouttaking the system offline. In one embodiment, nodes are configured usinga web interface to the master. Screenshots of example interfaces areillustrated with further reference to the example of FIG. 10.

In general, one or more masters can be implemented using one or moremachines, as illustrated in subsequent figures. The master 206 can beexecuted on the machine on which the collected dataset is received orgenerated or one or more different machines. Different types ofarchitectural configurations for the master 206 are illustrated insubsequent figures.

A receiving location, represented by data sink 203, can include, by wayof example, not limitation, a network connection, a physical disk, aconsole, a text file. The receiving location can also include, a file inthe Hadoop sequence file format, an HDFS path, a port, a monitoring oralerting application, and/or an IRC.

In general, the collected dataset from the data source 201 can beaggregated at the receiving location (data sink 203). The receivinglocation (e.g., or data sink 203) may further be mapped to node 204(e.g., a collector node), where the dataset collected from the datasetcan be aggregated. In some instances, analytics can be performed on thedataset upon collection at node 202 and/or aggregation at node 204. Inone embodiment, the master 206 controls the functions/properties of thenodes 202 and/or 204.

For example, the nodes 202/204 (e.g., logical nodes) can be associatedwith properties and/or functions, including but not limited to, addingencryption or digital signature, performing batching/unbatching,performing compression/uncompression, generating a checksum of adataset, performing sampling, performing benchmarking (e.g., byinjecting benchmark tags). In addition, the analytic that are performedcan include, feature extraction, filtering, transformation, generatingaggregate counts, statistics, etc.

The node 204, in aggregating the dataset, can also be mapped to a datasource 205 and a data sink 207, as controlled and configured at themaster 206. To aggregate data collected by the node 202, the data source205 of the node 204 can be mapped to the sink 203 of node 202.Additional functions and properties of nodes are illustrated withfurther reference to the example of FIG. 6-7.

In one embodiment, the node 204 (e.g., a collector node) writes theaggregated data to a storage location. The storage location can belocalized or distributed among multiple machines and the dataset isstored redundantly. The data sink 207 of the node 204 can be mapped to arepository 208, by the master 206, for example. The repository 208 maybe managed by a file system. The file system can be distributed (e.g.,the the Hadoop Distributed File System (HDFS)). Results of any analyticsperformed by nodes 202 and/or 204 can also be written to storage.Data/metadata extracted from the collected dataset may be written tostorage 208 as well.

Note that node 202, although illustrated in the example of FIG. 2A ashaving being mapped to a single node 204, can be mapped to multiplenodes, as further illustrated in the example of FIG. 2B.

FIG. 2B depicts diagrams showing examples of configuration of nodescontrolled/configured by a master and the dataflow therein from a datasource to a data sink.

In the example flow 215, an agent node 212 can send incomingdataset/event from a data source 211 to multiple collector nodes (e.g.,node 214 and node 216) which further maps the data flow to therepository 218. In one embodiment, the collector nodes are failovernodes for handling failures when appending new events. Failover sinkscan be used to specify alternate collectors to contact in the event theassigned collector fails. Fail over sinks can also be used to specify alocal disk sink to store data until the assigned collector recovers. Forexample, if node 214 goes down, the agent node 212 can deliver eventsvia collector agent 216 to the repository 218 or any other receivinglocation or destination. In some instances, the collector 216 can queuetheir logs until node 214 comes back online. The failover assignment canbe specified through the master 220. In addition, in one embodiment, thefailover mechanism is built in by default by disbursing collector nodeassignments evenly among agent nodes. This can mitigate the chances ofone collector from being overloaded when one collector fails.

Similarly, in the example flow 225, multiple collectors 224 are assignedto receive events/data flow from agent node 222. The collectors 224 canstore the event/data flow in the repository 228. The collectors 224 maybe failover chains. In addition, the collectors 224 can be fan outsinks, each of which receives the incoming event from the data source221. The fan out sinks can be configured by the master 230 as collectors224. The multiple collectors 224 as fan out sinks can be used for datareplication for processing data off a main data flow path.

FIG. 3A depicts a diagram showing one example configuration of a node(nodes) controlled by a master 306 residing on the same machine 300 anddataflow therein from a data source 301 to a data sink 305.

The agent node 302 and the master 306 used for controlling, configuring,and/or monitoring the node 302 can be implemented on the same machine300. The master 306 can be used to specify functions and/or propertiesof the agent node 302 in sending an incoming event/data set from thesource 310 to the sink 305.

In one embodiment, a collector node 304 is also implemented on the samemachine 304 as the agent node 302 and the master 306. The master 306 cancontrol, configure, and/or monitor both nodes 302 and 304 in mappingevents/data flow fro the source 301 to the sink 305.

FIG. 3B depicts a diagram showing example configurations of a node 312or nodes 312 and 314 executed on a machine 320 and controlled by amaster 316 which is executed on a different machine 330 and dataflowtherein from a data source 311 to a data sink 315.

The master 316 executed on machine 330 controls, configures, and/ormonitors the node 312 on the machine 320 in sending data sets/eventsfrom the data source 311 to the data sink 315. Alternatively, thecollector node 314 which is mapped to receive datasets/events from thenode 312 can also be executed on machine 320. The collector node can beconfigured to send datasets received from the node 312 to a data sink315. The master 316 can also control, configure, and/or monitor thecollector node 314.

However, having a single master is a single point of failure in thesystem. If the master 316 fails, the ability to update nodes (e.g.,nodes 314 and/or 314), configure nodes, control, and/or monitor nodes,etc. goes down. Thus, in one embodiment, multiple masters can beimplemented on multiple machines to maintain a consistent state in thesystem. In addition, using multiple masters on multiple machines allowsthe system to scale with increasing amount of log data and events toprocess and analyze.

Moreover, using multiple collectors can also increase log collectionthroughput and can improve timeliness of event/dataset delivery sincemore collector nodes are available at any given time during operation.Since data collection is parallelizable, the load from agent nodes canbe shared and distributed among many collector nodes. Examplearchitectures with multiple masters and multiple collectors areillustrated in FIG. 4-5.

FIG. 4 depicts a block diagram showing an example configuration of nodesin multiple tiers on the same machine 400 which controlled by multiplemasters 406 and 408 residing on different machines 420 and 430respectively, and dataflow therein from data sources 401, 403, and 405to a data sink 407.

The example system of FIG. 4 utilizes multiple agent nodes 402 in anagent tier 412 and multiple collector nodes 404 in a collector tier 414.In one embodiment, the nodes 402 in the agent tier 412 are located onthe machine 400 which is also producing logs that are being collected.For example, the data source 401 of the node 402 can be configured to bea port on the machine 400. The sink of the node 402 can be configured tobe node 404 in the collector tier 414.

In one embodiment, collector nodes 404 can listen and receive data frommultiple agent nodes 412 and aggregate logs. In addition, collectornodes 414 can further write the logs to a data sink 407 in the storagetier 416. In reliable mode agent nodes generally write data to therespective local disks. This data might not be deleted until after itknows that the data has reached its final destination. Collectorsgenerally don't write to their local disk—they actually forward data tothe storage tier which eventually points the collector to storage tiernodes that write to storage tier disks. An agent is generally on asystem that is responsible for doing other work (serving web pages) andthus agent nodes typically strive to have low resource utilization(e.g., cpu, memory, disk). Collector nodes are generally dedicated toprocessing large amounts of data and can use more expensive algorithmsbecause its processing units are generally less occupied with othertasks. Note that the data sink 407 may or may not be a storageunit/device. For example, the collector nodes 404 can also send the datasets downstream to cluster for further processing, a network, etc.

FIG. 5 depicts a diagram showing an example configuration of nodes 502and 504 in multiple tiers on different machines controlled by multiplemasters 506 residing on different machines and dataflow therein fromdata sources 501 to a data sink 507.

Agent nodes 502 can be executed on different machines and map incomingdata/events to collector nodes 504. The multiple machines may be in acluster and can include web servers. In general, the multiple machinesgenerate the log data to be collected and each of the multiple machinesis associated with at least one agent node to facilitate data collectionand/or aggregation. In one embodiment, the agent nodes 502 can forwardthe collected log data or events to a collector tier 512 which includesat least one collector node 504 but in some instances, multiplecollector nodes 504 which can combine streams of log data/events intolarger streams, for example, for writing to the storage tier 516. Thecollector node(s) 504 can aggregate the log data collected from themachines on which the agent nodes are executed. The collector nodes 504can also be executed on different machines with each machine havingvarious numbers of collector nodes 504, which passes events on to thedata sink 507.

In one embodiment, the collector node(s) 504 outputs the log data tomultiple destinations in the same or different formats. Some of thesupported formats include, by way of example not limitation, raw textfile, json (a web format), avro binary (an apache serialization format),a hadoop native sequence file, text emulating a syslog formatted entry,an apache web server formatted entry, and a “log4j” formatting pattern,etc. File formats are the format for batches of records. Record formatsare the formatting for individual events/records. A collector writing tostorage tier can be responsible for file formats. The work forformatting individual records however can be pushed upstream. As anexample, let's say the agent gets an event that has raw data An agentnode can extract features and/or reformat the data found in theevent/record and write this intermediate result to the metadata table.The downstream collector node can use the preformatted data and writeeach formatted record in the file format. One embodiment of the topologyincludes a storage tier 516 having a storage system coupled to themachines in the system. The storage system can include a distributedfile system to which the collector node(s) 504 in the collector tier 514stores the log data aggregated from the set of machines in the system.

Note that analytics can be performed on the log data/aggregated eventsby the agent nodes 502 and/or the collector nodes 504. The master 506can be used to control/configure the types of analytics that areperformed.

Agent nodes 502 and collector nodes 504 can be configured, re-configuredvia the masters 506, which can be executed on multiple machines that aredifferent from the machines that the collector 504 and agent nodes 502are executed on. In one embodiment, the masters 506 specify theconfiguration/topology of the system 500 and the agent node(s) 502 andthe collector node(s) 504 contact the master 506 to retrieveconfiguration information. For example, the master 506 can specify thatan agent data sink for the agent node 502 is a collector data source forthe collector node 504 in the collector tier 514 and that a collectordata sink 507 for the collector node 504 is the distributed file systemin the storage tier 516. In addition, the master 506 can be used by auser to configure or reconfigure the agent data sink and agent datasource and/or to configure or reconfigure the collector data source andthe collector data sink.

FIG. 6 depicts a block diagram illustrating example components of a node602 through which dataflow occur.

The node 602 can include a source specifier module 604, a sink specifiermodule 606, a configuration file 610, and/or a decorator module 620.

As used in this paper, a “module” or an “engine” includes a dedicated orshared processor and, typically, firmware or software modules that areexecuted by the processor. Depending upon implementation-specific orother considerations, an engine can be centralized or its functionalitydistributed. An engine can include special purpose hardware, firmware,or software embodied in a computer-readable medium for execution by theprocessor. As used in this paper, a computer-readable medium is intendedto include all mediums that are statutory (e.g., in the United States,under 35 U.S.C. 101), and to specifically exclude all mediums that arenon-statutory in nature to the extent that the exclusion is necessaryfor a claim that includes the computer-readable medium to be valid.Known statutory computer-readable mediums include hardware (e.g.,registers, random access memory (RAM), non-volatile (NV) storage, toname a few), but may or may not be limited to hardware.

One embodiment of the node includes a source specifier module 604 and asink specifier module 606. The source specifier module 604 and sinkspecifier module 606 can be implemented, example, as software embodiedin a computer-readable medium or computer-readable storage medium on amachine, in firmware, in hardware, in a combination thereof, or in anyapplicable known or convenient device or system. This and other modulesor engines described in this specification are intended to include anymachine, manufacture, or composition of matter capable of carrying outat least some of the functionality described implicitly, explicitly, orinherently in this specification, and/or carrying out equivalentfunctionality.

The source specifier module 604 can include an identifier or mapper to asource that produces or accepts events or data fed into the system. Thenode 604 reads or tails the event source. For example, the module 604can detect, recognize, identify, and/or store a port ID (e.g., a portname, a TCP or UDP port, port number) and a file ID (e.g., a file name,a file path, etc.). The file can be any type of file, including but notlimited to, messages, system log files, Hadoop log files, text files,etc. In addition, the event source can include, a console, a sequencefile (e.g., serialized events in a sequence file format), a source thatsynthetically generates a specified number of messages of a select size.

The sink specifier module 606 includes an identifier or mapper to alocation that consumes data or an event (e.g., a receiving ordestination location for data or events). A sink can include, by way ofexample, a disk, a repository, a distributed file system on a disk(e.g., the Hadoop distributed file system), a console, a device/machine,and/or a network/network connection over which the data/event is to beforwarded over. Thus, the sink specifier module 606 can detect,recognize, identify, and or store, a machine/device ID, a port ID, afile ID, and/or a path ID. In one embodiment, the path ID is adistributed sequence file path.

The source and sink specifier modules can store or retrieve theconfiguration information stored in the configuration file 610. Theconfiguration file 610 may be loaded with default values uponinstallation. In addition, the configuration file 610 may be updated bya master that controls the node 602. The updates the configuration file610 may occur while the system is running or when the system is off.

Note that the node 602 may be assigned to have an agent role or acollector rule. In general, an agent node is co-located onmachines/devices with the service/application that is generating,producing, creating events/logs/data. A collector node can receive,accept, check, listen for data from one or more agent nodes. A collectornode can be mapped to the data sink of an agent node, for example, toaggregate logs/data and/or to extract analytics. A collector node canfurther write the dataset, events, and/or any generated events tostorage (e.g., a distributed file system).

One embodiment of the node 602 includes a decorator module 620. Thedecorator module 620 can be implemented, example, as software embodiedin a computer-readable medium or computer-readable storage medium on amachine, in firmware, in hardware, in a combination thereof, or in anyapplicable known or convenient device or system. This and other modulesor engines described in this specification are intended to include anymachine, manufacture, or composition of matter capable of carrying outat least some of the functionality described implicitly, explicitly, orinherently in this specification, and/or carrying out equivalentfunctionality. The decorator module 620 can addproperties/functionalities of a sink and modify the data/event streamsthat pass through them, like a wrapper. For example, the decoratormodule 620 can be used to increase reliability, robustness of the systemor to perform analytics. An example of the decorator module isillustrated in detail with further reference to the example of FIG. 7. Adecorator can correspond to a sink that interposes between the sourceand the sink. The decorator processes events and then sends it down thepipeline to another sink (which may be another decorator wrappinganother sink) on the same node/dataflow. Sources produce events, andpass them through the decorator to sink.

FIG. 7 depicts a block diagram illustrating example components of adecorator module 720 in a node.

The decorator module 720 can include, a write ahead module 702, asampler module 704, an open module 706, an append module 708, abatch/unbatch module 710, and/or a GZIP/GUNZIP module 712.

As used in this paper, a “module” or an “engine” includes a dedicated orshared processor and, typically, firmware or software modules that areexecuted by the processor. Depending upon implementation-specific orother considerations, an engine can be centralized or its functionalitydistributed. An engine can include special purpose hardware, firmware,or software embodied in a computer-readable medium for execution by theprocessor. As used in this paper, a computer-readable medium is intendedto include all mediums that are statutory (e.g., in the United States,under 35 U.S.C. 101), and to specifically exclude all mediums that arenon-statutory in nature to the extent that the exclusion is necessaryfor a claim that includes the computer-readable medium to be valid.Known statutory computer-readable mediums include hardware (e.g.,registers, random access memory (RAM), non-volatile (NV) storage, toname a few), but may or may not be limited to hardware.

One embodiment of the decorator module 720 includes a write ahead module702. The decorator module 620 can be implemented, example, as softwareembodied in a computer-readable medium or computer-readable storagemedium on a machine, in firmware, in hardware, in a combination thereof,or in any applicable known or convenient device or system. This andother modules or engines described in this specification are intended toinclude any machine, manufacture, or composition of matter capable ofcarrying out at least some of the functionality described implicitly,explicitly, or inherently in this specification, and/or carrying outequivalent functionality.

The write ahead module 702 provides reliability and durability andwrites/stores events/datasets to persistent storage on the machinegenerating the event (writeAhead( . . . )) prior to forwarding them. Inthe event and any of the downstream nodes fails, the system has a copyof the dataset that can still be retrieved. In response to determiningthat the dataset/event has safely reached its destination, the copy canbe deleted from persistent storage. The write ahead module 702 can alsoassign batch identifiers for batches of messages/events to track

One embodiment of the decorator module 720 includes a sampler module704. The decorator module 704 can be implemented, example, as softwareembodied in a computer-readable medium or computer-readable storagemedium on a machine, in firmware, in hardware, in a combination thereof,or in any applicable known or convenient device or system. This andother modules or engines described in this specification are intended toinclude any machine, manufacture, or composition of matter capable ofcarrying out at least some of the functionality described implicitly,explicitly, or inherently in this specification, and/or carrying outequivalent functionality.

The sampler module 704 can sample data input or events at selectintervals (e.g., intervalSampler(N)). For example, the sample module 704can be configured to send every nth element/event from the source to thedata sink. In one embodiment, the sampler module 704 samples based on aprobability such that each event or message has a specified chance ofbeing forwarded (e.g., probSampler (p)). In addition, the sampler module704 can be a reservoir sampler (e.g., reserverSampler (K) such that whenflushed, a selected number (K) of elements are forwarded and each eventthat passes through has the same probability of being selected forforwarding.

One embodiment of the decorator module 720 includes an open module 706(or re-try open module). The open module 706 can be implemented,example, as software embodied in a computer-readable medium orcomputer-readable storage medium on a machine, in firmware, in hardware,in a combination thereof, or in any applicable known or convenientdevice or system. This and other modules or engines described in thisspecification are intended to include any machine, manufacture, orcomposition of matter capable of carrying out at least some of thefunctionality described implicitly, explicitly, or inherently in thisspecification, and/or carrying out equivalent functionality. In oneembodiment, the open module 706, in reading or opening a file orincoming event, continues to re-try the opening process if the initialattempt fails.

One embodiment of the decorator module 720 includes an append module 708(or re-try append module). The append module 708 can be implemented,example, as software embodied in a computer-readable medium orcomputer-readable storage medium on a machine, in firmware, in hardware,in a combination thereof, or in any applicable known or convenientdevice or system. This and other modules or engines described in thisspecification are intended to include any machine, manufacture, orcomposition of matter capable of carrying out at least some of thefunctionality described implicitly, explicitly, or inherently in thisspecification, and/or carrying out equivalent functionality. In oneembodiment, the append module 708, can append events to a target. In theevent that the append fails, the module 708 can close the target,re-open it and attempt to append again. In general, sinks and decoratormodules include open, append, and close functions. Typically, if one ofthese operations fail, the modules can give up after a first try. Usingthe open module (or re-try open module) or the append module (e.g.,re-try append module), the sink/decorator can be configured to keeptrying the failed operation (e.g.,retry opening or appending).

One embodiment of the decorator module 720 includes a batch/unbatchmodule 710. The batch/unbatch module 710 can be implemented, example, assoftware embodied in a computer-readable medium or computer-readablestorage medium on a machine, in firmware, in hardware, in a combinationthereof, or in any applicable known or convenient device or system. Thisand other modules or engines described in this specification areintended to include any machine, manufacture, or composition of mattercapable of carrying out at least some of the functionality describedimplicitly, explicitly, or inherently in this specification, and/orcarrying out equivalent functionality.

The batch/unbatch module 710 can buffer a select number ofevents/message together to be forwarded or sent as a single aggregateevent. The batch/unbatch module 710 can take an aggregate event andsplit and forward the original events.

One embodiment of the decorator module 720 includes a GZIP/GUNZIP module712. The GZIP/GUNZIP module 712 can be implemented, example, as softwareembodied in a computer-readable medium or computer-readable storagemedium on a machine, in firmware, in hardware, in a combination thereof,or in any applicable known or convenient device or system. This andother modules or engines described in this specification are intended toinclude any machine, manufacture, or composition of matter capable ofcarrying out at least some of the functionality described implicitly,explicitly, or inherently in this specification, and/or carrying outequivalent functionality. The GZIP/ZIP module 712 can zip or unzip anevent/message or multiple events/messages.

In general, multiple decorators can be assigned to an event, message, ordataset. For example, decorators can be combined to batch multiplemessages together and compress the messages before forwarding them tothe mapped data sink.

FIG. 8 depicts a block diagram illustrating example components of amaster 800 that controls the nodes in the system.

The master 800 can include a node configurator module 802, a systemhealth monitor 804, a consistency manager 806, a map module 808, and/oran acknowledger module 810. The master 800 can include additional orless modules. The master 800, although illustrated as being a discretecomponent, can be implemented on multiple machines, as shown in theexamples of FIG. 4-5 on full or partial redundancy.

The master 800, as implemented on a single or multiple machines canperform as a central control and monitor for the system. The master 800can track and monitor the status, health of nodes, and the data flow inthe system. In tracking and monitoring system status, the master 800 canautomatically perform any system re-configurations that may be needed toresolve node failure or load imbalance, for example.

The master 800 can identify, detect, or be used to specify new nodes ordeleted nodes in the system and perform any adjustments in data flow byupdating the mapping of data sinks and sources. In general, the nodescontact the master 800 to retrieve configuration information. When anode contacts the master, the node detects that the configurationversion has been changed and can instantiate and activate theconfiguration. This allows updates to the system and mapping to beperformed in dynamically, in operation, without starting or rebootingthe system. For example, a node's configuration can be updated duringoperation.

Additionally, a user can reconfigure the setting by accessing the master800. One of accessing the master 800 is through the master's web page.Examples of screenshots are illustrated in FIG. 10A-F. The configurationlink is displayed on the master's web interface. In the interface, forexample, node names are depicted and sources and sinks can be viewed,updated, or deleted. Once a configuration has been changed, the versionidentifier or stamp on the master's web page will be updated.

As used in this paper, a “module” or an “engine” includes a dedicated orshared processor and, typically, firmware or software modules that areexecuted by the processor. Depending upon implementation-specific orother considerations, an engine can be centralized or its functionalitydistributed. An engine can include special purpose hardware, firmware,or software embodied in a computer-readable medium for execution by theprocessor. As used in this paper, a computer-readable medium is intendedto include all mediums that are statutory (e.g., in the United States,under 35 U.S.C. 101), and to specifically exclude all mediums that arenon-statutory in nature to the extent that the exclusion is necessaryfor a claim that includes the computer-readable medium to be valid.Known statutory computer-readable mediums include hardware (e.g.,registers, random access memory (RAM), non-volatile (NV) storage, toname a few), but may or may not be limited to hardware.

One embodiment of the master includes a node configurator module 802.The node configurator module 802 can be implemented, example, assoftware embodied in a computer-readable medium or computer-readablestorage medium on a machine, in firmware, in hardware, in a combinationthereof, or in any applicable known or convenient device or system. Thisand other modules or engines described in this specification areintended to include any machine, manufacture, or composition of mattercapable of carrying out at least some of the functionality describedimplicitly, explicitly, or inherently in this specification, and/orcarrying out equivalent functionality.

In one embodiment, the node configurator module 802 detects, updates,identifies, stores configuration settings of the nodes in the system.For example, the source and sink mappers can configure the datasource(s) and/or data sink(s) for each node. In addition, the module 802can assign and/or track functions/properties (e.g., analytics functions)that a node can perform.

One embodiment of the master includes a system health monitor 804. Thesystem health monitor 804 can be implemented, example, as softwareembodied in a computer-readable medium or computer-readable storagemedium on a machine, in firmware, in hardware, in a combination thereof,or in any applicable known or convenient device or system. This andother modules or engines described in this specification are intended toinclude any machine, manufacture, or composition of matter capable ofcarrying out at least some of the functionality described implicitly,explicitly, or inherently in this specification, and/or carrying outequivalent functionality.

The system health monitor 804 can track, detect, determine, monitor,system traffic, node health/status, dataflow, load balance/imbalance inthe system. The master 800 can update/adjust data flow based on systemtraffic and any load imbalances. In addition, if certain nodes havefailed, the master 800 can assign failover nodes such that data flow isnot interrupted.

One embodiment of the master includes a consistency module 806. Theconsistency module 806 can be implemented, example, as software embodiedin a computer-readable medium or computer-readable storage medium on amachine, in firmware, in hardware, in a combination thereof, or in anyapplicable known or convenient device or system. This and other modulesor engines described in this specification are intended to include anymachine, manufacture, or composition of matter capable of carrying outat least some of the functionality described implicitly, explicitly, orinherently in this specification, and/or carrying out equivalentfunctionality.

The consistency module 806 maintains consistency among multiple mastersimplemented on multiple machines in a system, by leveraging theconsensus protocol and/or a gossip protocol. For example, the master 800can maintain consistency using a consensus protocol for order criticalstate changes and using a gossip protocol for weak consistency amongdata including but not limited to node status updates, reportinginformation, and metrics information.

One embodiment of the master includes a map module 808 and anacknowledger module 810. The map module 808 and the acknowledger module810 can be implemented, example, as software embodied in acomputer-readable medium or computer-readable storage medium on amachine, in firmware, in hardware, in a combination thereof, or in anyapplicable known or convenient device or system. This and other modulesor engines described in this specification are intended to include anymachine, manufacture, or composition of matter capable of carrying outat least some of the functionality described implicitly, explicitly, orinherently in this specification, and/or carrying out equivalentfunctionality.

The map module 808 monitors and manages a set of tags associated withbatches of messages to track which messages or batches or messages havebeen successfully sent to a receiving location or stored in a datarepository.

In one embodiment, an agent node generates a batch comprising multiplemessages from the log data and assigns a tag (e.g., can include a beginevent tag and an end event tag) to the batch. The agent can also computea checksum for the batch of multiple messages and stores he tag and thechecksum to a write-ahead-log (WAL) in persistent storage of the machinefrom which the dataset was generated.

The agent node sends the batch of messages to a collector which verifiesthe checksum of the batch, and if successful, adds the tag to a map oftags for batches of messages. When the collector node determines that abatch of messages has successfully reached a receiving location orstored in storage/file system, the collector agent publishes the tag tothe map module 808 of the master. Thus, the map module 808 includesidentifiers for events, messages, or batches of messages that havereached their mapped locations. The master 800 can use the identifiersin the map module 808 to notify the agent nodes of batches that havebeen successfully stored or sent. In one embodiment, the acknowledgermodule 810, using the tags that are associated with batches in the map,acknowledges the agent node and the collector node of the batches thathave been safely stored in the file system or sent down stream.

FIG. 9 depicts an example of an extensible data model used to representevents or data flow in a distributed system from a source to a sink.

The data model, for example, can include one or more timestamp fields902 and 904, a priority field 906, a source host field 908, a body field910, and/or a metadata table field 912. The data model is extensible andthus additional fields can be added on the fly during data flow. Thedata model is described with further reference to the example of FIG.15.

FIG. 10A depicts a screenshot of an example user interface showing nodestatus and configurations with node mappings. This user interface can beaccessed from the master. FIG. 10B depicts a screenshot of an exampleuser interface for configuring a node. The configuration page can beaccessed from the master's web page. For example, a node to beconfigured can be selected from the list and its source/sink can bespecified.

FIG. 10C depicts a screenshot of an example user interface forconfiguring multiple nodes. FIG. 10D depicts a screenshot of an exampleuser interface for issuing raw commands. Commands can be used to modifythe global state of the system. For example, the “config” command setsthe configuration of a flow on the master, who in turn sends the info toa node. The “refresh” command takes an existing config and forces thenode to reset (close then open). The “decommission” command removes alogical node. The “unmap” command removes a logical node from a physicalnode. The “spawn” command creates a new logical node (could be used toassociate an unmapped logical node onto another physical machine).Confirmation that command was received by the master by reviewing thecommand history section of FIG. 10A. Confirmation that a node hasinstalled the changes the command may have caused by looking at the webpage in the examples of FIGS. 10E and 10F. FIG. 10E depicts a screenshotof an example user interface showing statistics for a node. The userinterface shows the data flow configuration settings, and memory usageof a particular node. For example, this figure shows information aboutthe node “LogicalNode[0]” and includes two analytics reports: a graphshowing the number of incoming events and a graph showing a histogram ofthe frequency of certain words in the stream. The source in this exampleis Twitter, and the sinks for the node include the graphs. FIG. 10Fdepicts a screenshot of an example user interface showing statistics fora node. This figure illustrates the configuration info for this dataflow, and similarly has an event graph and a histogram (showing an evendistribution of substrings).

FIG. 11 depicts a flowchart of an example process of facilitatingcollecting and aggregating datasets that are machine or user-generatedfor analysis.

In process 1102, a dataset is collected on a machine on which thedataset is received or generated. The data set can include logs from anapplication that is local or remote, a service, a social networkingsite, feeds, status updates, etc. The dataset can be collected from adata source on the machine. In addition, datasets can be collected frommachines in a cluster which can include web servers, for example. Inprocess 1104, the dataset that is collected from the data source isaggregated at a receiving location. The receiving location can be, forexample, a network, a network connection, a physical disk, a consol, atext file, a file in a sequence file format, an HDFS path, a port, amonitoring or alerting application, and/or an IRC. One or more receivinglocation can be specified.

In one embodiment, the mapping of the data source to the receivinglocation is controlled by a master. The master can be executed on adifferent machine than the machine on which the dataset is generated orreceived. The master can also be executed on the same machine as the oneon which the dataset is received or generated. In one embodiment, thedata source is mapped to a receiving location via a logical node. Thelogical node can contact the master to obtain configuration informationincluding but not limited to, functions, data source, data sink, etc. Inprocess 1106, analytics are performed on the dataset upon collection oraggregation. Analytics that can be performed include, by way of example,feature extraction, filtering, and transformation, generation ofaggregate counts, and/or generation of statistics. The analytics can beperformed in near real time relative to collection of the dataset. Inprocess 1108, the dataset aggregated at the receiving location iswritten to a storage location. With IO there is typically a tradeoffbetween throughput and latency. Usually, sending lots of small thingswill take a lot longer than a few large things, even if the actualamount of payload is the same. This is generally due to overheadassociated with each sent value (sent events in Flume's case). In oneembodiment, delay (latency) can be added to help the scheduler improveon throughput.

The storage location can be distributed among multiple machines and thedataset is stored redundantly on a distributed file system

FIG. 12 depicts a flowchart of an example process for configuring asystem to collect and aggregate datasets.

In process 1202, a client (e.g., a web page/interface or a command lineinterface, another program (a master), or a device (e.g., a laptop, acell phone, a smartphone, iPhone, etc.) is launched on a machine toaccess a master for the system, for example, using the master'suniversal resource locator (URL). The master's web page can be used toview, change, and/or update a node's configuration, as illustrated inthe example of FIG. 10A. The system may include multiple machines (e.g.,in a cluster) which generate datasets that are to be collected. Inprocess 1204, a data source in the system from where dataset is to becollected is identified.

In process 1206, a machine in the system that generates the dataset tobe collected is configured to send the dataset to the data source. Inprocess 1208, an arrival location where the dataset that is collected isto be aggregated or written is identified. In process 1210, an agentnode is configured by specifying a source for the agent node as the datasource in the system and specifying a sink for the agent node as thearrival location. In addition, the agent node is generally configurableto perform analytics on the dataset.

In process 1212, the arrival location is specified as a collector sourceof a collector node. In process 1214, a distributed file system isspecified as a collector sink of the collector node. The distributedfile system can be, for example, the Hadoop distributed file system(HDFS)

FIG. 13 depicts a flowchart of an example for scaling the system byimplementing multiple masters on multiple machines.

A master provides a centralized management point to monitor and updatedata flows. Changes in the system are told to the master, and changescan come from the master. When new resources are added to the system,the information is updated at the master. In addition, if a user issuesa command that changes global state, the master can be used such thatthe master can react to these state changes. In addition, the master candynamically respond to system changes such as load imbalances, nodefailure, machine failure, new hardware, etc. For example, a userconfigures a new node to be a collector. The master detects this, andautomatically reacts by changing the configuration of some nodes to sendevents downstream to the new collector. The master can do this bychanging global state. In another example, if a collector node hasn'tcommunicated with the master for a while. The master may decide that themachine might have fialed and automatically reacts by sending an eventto an admin (via email/irc/im/sms, etc.). Similarly, in yet anotherexample, nodes periodically report their performance metrics to themaster (e.g., in the examples of FIGS. 10E, 10F). The reports indicatethat a collector is being overloaded with work, and reacts to newmetrics data by shifting some of the work from the overloaded collectorto another with less load. This can be performed by changing globalstate.

Nodes in the system can be configured/reconfigured dynamically via themaster. The nodes can be arranged in a three-tier topology (e.g., agenttier, collector tier, storage tier) or any other types of topology withany number of tiers. A user can use the master to configure the topologyand the nodes, for example, by accessing a link on the web page for themaster, as illustrate in the example screenshot of FIG. 10B.

Since the master provides centralized management, having only one masteris a single point of failure in the system. Thus, in one embodiment, inprocess 1302, the system is scaled by implementing multiple masters onmultiple machines.

In process 1304, consistency is maintained among the multiple mastersusing a consensus protocol for order critical state changes. Forexample, order critical state changes can include data flowconfigurations and node-to-machine mappings. In process 1306,consistency is maintained among the multiple masters using a gossipprotocol. For example, node status updates, reporting information, andmetrics information are maintained using the gossip protocol. Typically,stronger consistency is maintained using the consensus protocol andweaker consistency is maintained using the gossip protocol.

FIG. 14 depicts a flowchart of another example process for configuring asystem to collect and aggregate datasets and to perform analytics on thedatasets.

In process 1402, data sources on the multiple machines wherein datasetsare to be collected from are identified. In process 1404, the multiplemachines in the system that generate the datasets to be collected areconfigured to send the datasets to the data source. In process 1406, anarrival location where dataset that is collected is to be logged isidentified.

In process 1408, configurations for the multiple machines are specifiedsimultaneously by accessing a master through a web page and specifyingthe data sources for agent nodes. The master can be used to configureand/or dynamically reconfigure the agent nodes. In addition, statuses ofthe agent nodes are visible via the master.

In process 1410, a sink is specified for each of the agent node as thearrival location. In one embodiment, the sink is configured to be afan-out sink that sends incoming datasets to child sinks coupled to thesink for data replication or for splitting off a CPU intensive tap. Inaddition, the sink can be configured as a failover sink that specifiesalternate collectors in case the collector fails. For example, thefailover sink can specify a local disk to store data until the sink thathas failed recovers.

In process 1412, the arrival location is specified as a collector sourceof a collector node. In process 1414, a distributed file system isspecified as a collector sink of the collector node.

FIG. 15 depicts a flowchart of an example process for dynamicallyprocessing an event including a dataset that is streamed from a sourceto a sink via nodes.

Events or messages are transmitted in a flexible and extensible datamodel. For example, events can be formatted as unstructured blobs andcan be annotated with key-value attribute pairs which can be arbitrary.This allows analytic features include feature extraction, filtering, andtransformation to be performed as the system forwards datasetsdownstream. The general pattern can utilize use a feature extractor toadd key-value attributes, and then use a analysis module that readskey-value attributes and acts on it. There are feature extractionmodules in the system, that can scan raw data and pull out structureddata (e.g., numbers). This data can be written as a key-value attribute.For example raw data can take upon the form: “[31 Mar. 2010, 12:06 AM]Jon wrote this example”. The system can use a regular expression to pullout the data and turn it into a single number representation (unixtime). The system can add a metadata attribute called ‘eventTime’ withthe numerical representation as its value.

Downstream, there could be a module that looks at this value andcompares it against a threshold predicate. For example, a thresholdpredicate can be: “was ‘eventtime’ before 31 Mar?” Alternately, ahistogramming module can use that value and determine the number ofevents that were on 31 Mar (and 30 Mar, and 1 Apr, etc). The examplegraphs shown in the examples of FIGS. 10E and 10F can be generated usingthis pattern.

In process 1502, attributes of the event is specified in a data model.An example of the data model is illustrated in FIG. 9. In oneembodiment, the data model is extensible to add properties to the eventas the dataset is streamed from the source to the sink. In addition,additional data fields can be appended to the data model in response toadditional sources being added to the event. The attributes can includea priority, which can be specified by a logging system. If a machine isheavily loaded, some events can be prioritize over others, using the‘priority field’. For example, FATAL (highest priority) messages mayskip ahead of line and be sent before lesser prioritized messages (e.g.,ERROR or WARN priority messages). Also, if the system is, for example,running out of disk space, the DEBUG priority data can be erased beforeany of the INFO data gets erased, and INFO before WARN, etc. Inaddition, the attributes can include a source machine, a body, a timestamp, a metadata table, etc. Events will generally include each ofthese attributes but the body may have zero length and the metadatatable may be empty.

In one embodiment, the timestamp can be a UNIX timestamp which is thetime stamp from the source machine. The UNIX timestamp can be measuredin milliseconds, for example. The timestamp may be a nanosecondtimestamp, or any other monotonically increasing identifier which is amachine specific counter from the source machine. The priority field ofthe message can take on example values: TRACE, DEGBUG, INFO, WARN,ERROR, OR FATAL. In general, a logging system (e.g., syslog or log4j)provides the priority values. The source host can include a machine nameor IP address. The body can include the raw log entry body. By default,the body can be truncated to a max of 32 KB per event. The max size canbe reconfigured.

In one embodiment, event/message output to directories or files can becontrolled based on the values of data model representing the event. Forexample, the system or any analytics being performed uses the timestampto bucket the dataset based on time or date, specified by day of week,month, date and time, day of month, hour, minute, locale's equivalent ofAM or PM, seconds, seconds since, year, etc. For example, whenmonitoring a web server, logs can be bucketed based on time, page hit,or the browser that is used, using extraction mechanisms. When trackingnodes, logs can be bucketed based on time and data node name, forexample. In addition, the data group can be grouped based on a projectidentifier or person, for example, when tracking a feed of JIRA ticketsfrom an apache feed.

In process 1504, the event described by the data model is annotated withkey-value attribute pairs in a metadata table to enable analyticfeatures to be performed on the dataset. In one embodiment, the metadatatable includes a map from a string attribute name to an array of bytes.The metadata table can be used to specify routing of the event tomultiple or different storage locations. In addition, the metadata tablecan be used to specify outputting of the dataset at the sink in multipleor different formats. The acktags, rolltags, and checksum tags/valuesare examples of values that use the metadata table. Metrics informationgenerated by nodes can use the metadata table to store individualmetrics. Any additional features can also utilize tag/values and themetadata table to allow new fields without rebuilding the program).

In process 1506, an event including a dataset that is streamed from asource to a sink via nodes is dynamically processed.

FIG. 16 depicts a flowchart of an example process for collecting andaggregating datasets for storage in a file system with fault tolerance.

In process 1602, datasets are collected from a data source on a machinewhere the datasets are generated. The data can be collected from anagent node on the machine. In process 1604, a batch ID comprisingmultiple messages is generated from the datasets. In one embodiment, theagent node generates the batch ID. In process 1606, a tag is assigned tothe batch ID and a checksum is computed for the batch, for example, alsoby the agent node.

In process 1608, the batch ID and messages are written to an entry in awrite-ahead-log (WAL) in storage. The writing can be performed by theagent node. In one embodiment, the storage for the WAL is on the machinewhere the datasets are generated. In process 1610, the datasets are sentto a receiving location (e.g., a collector node). The data set on theagent can be sent when the size or time elapsed reaches a certainthreshold.

In process 1612, a file is written to the destination location, forexample, by the collector node. In process 1614, the tag is added to amap, in response to verifying the checksum of the batch of multiplemessages at a known location (e.g., the master). The map can beassociated with multiple tags assigned to multiple batches of messagesfrom the datasets. In process 1616, tags associated with the batches inthe file that have been written to the destination location areidentified in the map. The process continues in the flow chart of FIG.17.

FIG. 17 depicts a flowchart of an example process for deleting the WALin response to acknowledgement of receipt by the master.

In process 1702, an acknowledgement message that the batch has beenwritten to the destination location is queried for. In process 1704, itis determined whether the batch has been written in the destinationlocation, which can be a storage tier comprising a distributed filesystem. If so, in process 1706, the entry is deleted from the storage ofthe machine. If not, in process 1708, the batch is resent.

FIG. 18 depicts a flowchart of an example process for collecting andaggregating datasets with fault tolerance using a store on failuremechanism.

In process 1802, a dataset is collected from a data source on a machinewhere the dataset is generated.

In process 1804, the dataset is sent to a receiving location whichaggregates the dataset. In one embodiment, the receiving location is acollector in a collector tier on the machine where the dataset isgenerated or another machine. The receiving location may also be a filesystem in a storage tier comprised in one or more storage devices.

In process 1806, the dataset is stored in persistent storage of themachine until the receiving location has been repaired or until anotherdestination is identified in response to determining that receivinglocation which is mapped to receive the dataset has failed.

FIG. 19 shows a diagrammatic representation of a machine 1900 in theexample form of a computer system within which a set of instructions,for causing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

In the example of FIG. 19, the computer system 900 includes a processor,memory, non-volatile memory, and an interface device. Various commoncomponents (e.g., cache memory) are omitted for illustrative simplicity.The computer system 900 is intended to illustrate a hardware device onwhich any of the components depicted in the example of FIG. 1 (and anyother components described in this specification) can be implemented.The computer system 900 can be of any applicable known or convenienttype. The components of the computer system 900 can be coupled togethervia a bus or through some other known or convenient device.

The processor may be, for example, a conventional microprocessor such asan Intel Pentium microprocessor or Motorola power PC microprocessor. Oneof skill in the relevant art will recognize that the terms“machine-readable (storage) medium” or “computer-readable (storage)medium” include any type of device that is accessible by the processor.

The memory is coupled to the processor by, for example, a bus. Thememory can include, by way of example but not limitation, random accessmemory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). Thememory can be local, remote, or distributed.

The bus also couples the processor to the non-volatile memory and driveunit. The non-volatile memory is often a magnetic floppy or hard disk, amagnetic-optical disk, an optical disk, a read-only memory (ROM), suchas a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or anotherform of storage for large amounts of data. Some of this data is oftenwritten, by a direct memory access process, into memory during executionof software in the computer 900. The non-volatile storage can be local,remote, or distributed. The non-volatile memory is optional becausesystems can be created with all applicable data available in memory. Atypical computer system will usually include at least a processor,memory, and a device (e.g., a bus) coupling the memory to the processor.

Software is typically stored in the non-volatile memory and/or the driveunit. Indeed, for large programs, it may not even be possible to storethe entire program in the memory. Nevertheless, it should be understoodthat for software to run, if necessary, it is moved to a computerreadable location appropriate for processing, and for illustrativepurposes, that location is referred to as the memory in this paper. Evenwhen software is moved to the memory for execution, the processor willtypically make use of hardware registers to store values associated withthe software, and local cache that, ideally, serves to speed upexecution. As used herein, a software program is assumed to be stored atany known or convenient location (from non-volatile storage to hardwareregisters) when the software program is referred to as “implemented in acomputer-readable medium.” A processor is considered to be “configuredto execute a program” when at least one value associated with theprogram is stored in a register readable by the processor.

The bus also couples the processor to the network interface device. Theinterface can include one or more of a modem or network interface. Itwill be appreciated that a modem or network interface can be consideredto be part of the computer system 1900. The interface can include ananalog modem, isdn modem, cable modem, token ring interface, satellitetransmission interface (e.g. “direct PC”), or other interfaces forcoupling a computer system to other computer systems. The interface caninclude one or more input and/or output devices. The I/O devices caninclude, by way of example but not limitation, a keyboard, a mouse orother pointing device, disk drives, printers, a scanner, and other inputand/or output devices, including a display device. The display devicecan include, by way of example but not limitation, a cathode ray tube(CRT), liquid crystal display (LCD), or some other applicable known orconvenient display device. For simplicity, it is assumed thatcontrollers of any devices not depicted in the example of FIG. 19 residein the interface.

In operation, the computer system 1900 can be controlled by operatingsystem software that includes a file management system, such as a diskoperating system. One example of operating system software withassociated file management system software is the family of operatingsystems known as Windows® from Microsoft Corporation of Redmond, Wash.,and their associated file management systems. Another example ofoperating system software with its associated file management systemsoftware is the Linux operating system and its associated filemanagement system. The file management system is typically stored in thenon-volatile memory and/or drive unit and causes the processor toexecute the various acts required by the operating system to input andoutput data and to store data in the memory, including storing files onthe non-volatile memory and/or drive unit.

Some portions of the detailed description may be presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, transferred, combined, compared, and otherwisemanipulated. It has proven convenient at times, principally for reasonsof common usage, to refer to these signals as bits, values, elements,symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the methods of some embodiments. The requiredstructure for a variety of these systems will appear from thedescription below. In addition, the techniques are not described withreference to any particular programming language, and variousembodiments may thus be implemented using a variety of programminglanguages.

In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in a client-server network environment, or as a peermachine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a laptop computer, a set-top box (STB), apersonal digital assistant (PDA), a cellular telephone, an iPhone, aBlackberry, a processor, a telephone, a web appliance, a network router,switch or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

While the machine-readable medium or machine-readable storage medium isshown in an exemplary embodiment to be a single medium, the term“machine-readable medium” and “machine-readable storage medium” shouldbe taken to include a single medium or multiple media (e.g., acentralized or distributed database, and/or associated caches andservers) that store the one or more sets of instructions. The term“machine-readable medium” and “machine-readable storage medium” shallalso be taken to include any medium that is capable of storing, encodingor carrying a set of instructions for execution by the machine and thatcause the machine to perform any one or more of the methodologies of thepresently disclosed technique and innovation.

In general, the routines executed to implement the embodiments of thedisclosure, may be implemented as part of an operating system or aspecific application, component, program, object, module or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions set at various times invarious memory and storage devices in a computer, and that, when readand executed by one or more processing units or processors in acomputer, cause the computer to perform operations to execute elementsinvolving the various aspects of the disclosure.

Moreover, while embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that the various embodiments are capable of beingdistributed as a program product in a variety of forms, and that thedisclosure applies equally regardless of the particular type of machineor computer-readable media used to actually effect the distribution.

Further examples of machine-readable storage media, machine-readablemedia, or computer-readable (storage) media include but are not limitedto recordable type media such as volatile and non-volatile memorydevices, floppy and other removable disks, hard disk drives, opticaldisks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital VersatileDisks, (DVDs), etc.), among others, and transmission type media such asdigital and analog communication links.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof, means any connection or coupling,either direct or indirect, between two or more elements; the coupling ofconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import, when used in this application, shall referto this application as a whole and not to any particular portions ofthis application. Where the context permits, words in the above DetailedDescription using the singular or plural number may also include theplural or singular number respectively. The word “or,” in reference to alist of two or more items, covers all of the following interpretationsof the word: any of the items in the list, all of the items in the list,and any combination of the items in the list.

The above detailed description of embodiments of the disclosure is notintended to be exhaustive or to limit the teachings to the precise formdisclosed above. While specific embodiments of, and examples for, thedisclosure are described above for illustrative purposes, variousequivalent modifications are possible within the scope of thedisclosure, as those skilled in the relevant art will recognize. Forexample, while processes or blocks are presented in a given order,alternative embodiments may perform routines having steps, or employsystems having blocks, in a different order, and some processes orblocks may be deleted, moved, added, subdivided, combined, and/ormodified to provide alternative or subcombinations. Each of theseprocesses or blocks may be implemented in a variety of different ways.Also, while processes or blocks are at times shown as being performed inseries, these processes or blocks may instead be performed in parallel,or may be performed at different times. Further any specific numbersnoted herein are only examples: alternative implementations may employdiffering values or ranges.

The teachings of the disclosure provided herein can be applied to othersystems, not necessarily the system described above. The elements andacts of the various embodiments described above can be combined toprovide further embodiments.

Any patents and applications and other references noted above, includingany that may be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the disclosure can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further embodiments of thedisclosure.

These and other changes can be made to the disclosure in light of theabove Detailed Description. While the above description describescertain embodiments of the disclosure, and describes the best modecontemplated, no matter how detailed the above appears in text, theteachings can be practiced in many ways. Details of the system may varyconsiderably in its implementation details, while still beingencompassed by the subject matter disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the disclosure should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the disclosure with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the disclosure to the specific embodimentsdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe disclosure encompasses not only the disclosed embodiments, but alsoall equivalent ways of practicing or implementing the disclosure underthe claims.

While certain aspects of the disclosure are presented below in certainclaim forms, the inventors contemplate the various aspects of thedisclosure in any number of claim forms. For example, while only oneaspect of the disclosure is recited as a means-plus-function claim under35 U.S.C. §112, ¶113, other aspects may likewise be embodied as ameans-plus-function claim, or in other forms, such as being embodied ina computer-readable medium. (Any claims intended to be treated under 35U.S.C. §112, ¶113 will begin with the words “means for”.) Accordingly,the applicant reserves the right to add additional claims after filingthe application to pursue such additional claim forms for other aspectsof the disclosure.

1. A method for configuring a system to collect and aggregate datasets,the method, comprising: identifying a data source in the system fromwhere dataset is to be collected; configuring a machine in the systemthat generates the dataset to be collected, to send the dataset to thedata source; identifying an arrival location where the dataset that iscollected is to be aggregated or written; configuring an agent node byspecifying a source for the agent node as the data source in the systemand specifying a sink for the agent node as the arrival location.
 2. Themethod of claim 1, wherein, the system includes multiple machines whichgenerate datasets that are to be collected.
 3. The method of claim 1,further comprising: specifying the arrival location as a collectorsource of a collector node; specifying a distributed file system as acollector sink of the collector node; wherein, the distributed filesystem is the Hadoop distributed file system (HDFS)
 4. The method ofclaim 1, wherein, the agent node is configurable to perform analytics onthe dataset.
 5. The method of claim 1, further comprising, launching aweb page on a machine to access a master for the system using themaster's universal resource locator (URL).
 6. The method of claim 5,further comprising, scaling the system by implementing multiple masterson multiple machines.
 7. The method of claim 6, further comprising,maintaining consistency among the multiple masters using a consensusprotocol for order critical state changes.
 8. The method of claim 7,wherein, the order critical state changes include data flowconfigurations and node-to-machine mappings.
 9. The method of claim 6,further comprising, maintaining consistency among the multiple mastersusing a gossip protocol.
 10. The method of claim 9, wherein, the gossipprotocol is used to maintain weak consistency among data including butnot limited to node status updates, reporting information, and metricsinformation.
 11. The method of claim 5, wherein, configuration of theagent node is performed by accessing a link on the web page for themaster.
 12. The method of claim 1, wherein, the data source includes,one or more of, a syslog port, an incoming network connection, and anIRC channel.
 13. The method of claim 1, wherein, the data sourceincludes, a static file source.
 14. The method of claim 1, wherein, thedata source includes, one or more of, a status update and a feed. 15.The method of claim 1, wherein, the data source includes, output fromexecution of a program.
 16. The method of claim 1, wherein, the datasink includes a network connection.
 17. The method of claim 1, wherein,the data sink includes a physical disk.
 18. The method of claim 1,wherein, the data sink includes a console.
 19. A method for configuringa system having multiple machines to collect datasets from the multiplemachines and to perform analytics on the datasets, the method,comprising: identifying data sources on the multiple machines whereindatasets are to be collected from; configuring the multiple machines inthe system that generate the datasets are to be collected, to send thedatasets to the data source; identifying an arrival location wheredataset that is collected is to be logged; specifying configurations forthe multiple machines simultaneously by accessing a master through a webpage and specifying the data sources for agent nodes; wherein, eachagent node is associated with each of the multiple machines; specifyinga sink for each of the agent node as the arrival location; specifyingthe arrival location as a collector source of a collector node;specifying a distributed file system as a collector sink of thecollector node; wherein, the agent node or the collector node performsanalytics on the datasets.
 20. The method of claim 19, wherein, themaster is used to configure and dynamically reconfigure the agent nodes.21. The method of claim 19, wherein, statuses of the agent nodes arevisible via the master.
 22. The method of claim 19, wherein, thedistributed file system is the Hadoop distributed file system (HDFS).23. The method of claim 19, further comprising, launching a web page ona machine to access the master using the master's universal resourcelocator (URL); wherein, configuration of the agent nodes for themultiple machines is performed by accessing a link on the web page forthe master.
 24. The method of claim 19, further comprising, configuringthe sink to be a fan-out sink that sends incoming datasets to childsinks coupled to the sink for data replication or for splitting off aCPU intensive tap.
 25. The method of claim 19, further comprising,configuring the sink as a failover sink that specifies alternatecollectors in case the collector fails.
 26. The method of claim 25,wherein, the sink is a failover sink that specifies a local disk tostore data until the sink that has failed recovers.